A way that may be employed to efficiently cool data centers and electronics housed in electronics which employs Loop Heat Pipes and other passive technologies to cool the primary heat loads in such systems and which may employ a combination of other methods to cool the secondary heat loads.
The cooling of electronics housed in enclosures has for many years been dominated by methods that were much more concerned about getting the job done than the energy it took to get the job done. Methods for improving the efficiency of electronic cooling using passive heat transfer such as heat pipes have been available since at least the Manhattan project, yet have only become inexpensive with the advent of CPUs that rejected 40 or more Watts, and needed to extend the operating capabilities of finned heat sinks. This disclosure employs passive closed loop heat transfer devices that can dramatically improve not only the energy efficiency of electronic cooling but also makes it possible to cool devices that reject 500 or more Watts mounted on densely packed printed circuit boards and in the case of a data center reducing the energy required to cool it by 80% or more!
The devices at the heart of this disclosure are Loop Heat Pipes, Capillary Pumped Loops and derivatives of Loop Heat Pipes that included devices like pumps in the condenser lines. We lump all these devices together into the category Loop Heat Pipe Like (LHPL).
In general the LHPLs employed provide the best energy efficiency of any electronic cooling device ever invented. Not only are they passive, but their ability to reject heat to new locations is often measured in meters employing very small condenser pipes (often less than 3 mm) that make it possible for them to move heat out of tight spaces to condensers that can reject the heat over large heat transfer areas to secondary coolants such as air and water. And it is this ability to transport heat to new locations in a chassis or enclosure and then pass it through a reasonably long pipe in a condenser which distributes it over a reasonably large area, that makes it possible to efficiently transfer the heat being rejected by tiny hot spots to secondary coolants which in turn remove the heat to cooling loops which ultimately reject it to the outside world. And it is also this ability to distribute the primary heat load over large areas that makes it possible to create very efficient counter flow heat exchangers that retain the quality of the heat as well, that makes this technology so exciting. In fact, the secondary coolants that it heats up are produced with the highest delta T's of any technology we know of. It is the low total thermal resistances of these devices that make it possible to produce LHPLs whose overall thermal resistance is only 0.15.degree. C./Watt and that have a heat transfer coefficient of 0.15.degree. C./(Wcm.sup.2). In the case of a 100 Watt CPU whose LHP condenser was cooled by water at 30.degree. C. the output from the condenser turned out to be 47.degree. C. with the CPU running at a heat spreader temperature of 59.degree. C. Translated into practical terms, employed in the 1 U rack mount chassis that dominate modern data centers, this device makes it possible to remove the heat directly from a server housed in a rack cabinet and move it directly back to the cooling tower in the data center. In the process, the noisy fans in the racks that produce many points of failure and can consume as much as 30% of the energy being used by servers along with the main CRAC unit blowers and water chillers that consume 35% of the total power employed by the data center end up getting cut out of the cooling process. The temperature we chose to cool these LHPs with, 30.degree. C., was chosen based on ASHRAE tables, and the performance of commercially available evaporative cooling towers. This temperature turns out to be the temperature of the coolant that this type of cooling tower will produce running in Atlanta Ga. on the hottest and most humid day of the year. A quick comparison of the power consumption at institutions like Lawrence Livermore National Labs suggests that:
TABLE 1 Electronics 50% Water Chiller 25% Air Blower 10% 1U fans 9% UPS 5% Lighting 1% changes to this:
TABLE-US-No. 00002 Electronics 83% 1U fans 1.6% Cooling Tower Pump and fan 5% UPS 8.3% Lighting 1.6%
Which is to say the total power consumed by the data center goes down by 40%!
The same energy benefits that accrue to data center cooling also accrue to the general cooling of all electronic enclosures that are air cooled, but to a lesser extent, for the simple reason that air is a much poorer heat transport medium than the chilled water that gets employed to move heat from servers that are cooled with it back to the water chiller or in the best case cooling case we have run up against, the data center cooling tower. At the head of the list of benefits in addition to reduced energy costs are huge reductions in noise, the elimination of heat arriving at the walls of enclosures that can be so hot that it is almost possible to get burned touching them, the frequent failure of rotating cooling components including fans and pumps (in the case of pumped liquid cooling) which now occur so often that the systems that employ them have to mount them so that they can be easily swapped out without turning the machine off along with the ability to reject heat loads from devices that produced 500 or more Watts and to cool efficient devices such as CPUs and GPUs mounted in laptops where improved energy efficiency can improve battery life.
To appreciate the benefits of employing LHPLs to cool electronic enclosures, including air and water cooled electronic devices used to do everything from control the operation of space vehicles to reject heat to the cooling towers of data centers, it is first necessary to recite the goals of this disclosure, which were to efficiently cool electronic enclosures in which semiconductor devices that rejected large quantities of heat (greater than 200 Watts) mounted on densely packed PCBs along with devices that shared the same enclosures that rejected the balance of the heat but did not provide a dense source of heat. In the case of air cooled enclosures housed in rack mounted chassis, we wanted to make the first goal achievable while at the same time improving the quality of heat being rejected to the data centers CRAC system. In cases where liquid cooling, including chilled water was available on the data center floor, our goal was to reject high quality heat all the way back to the data center cooling tower, on a year round basis in most localities in the world. To achieve these goals, we employed LHPLs, some of whose other outstanding properties in addition to the fact that they are passive devices, turns out to be that the eliminate most of the electric motors, fans, blowers, compressors and other rotating devices found in servers and throughout the data center that end up making noise, contribute to frequent server failures and cost money to maintain and operate.
These goals don't get met without skepticism from prior art and other technologies, so we will now address the advantages of our approach in detail, while at the same time laying out the critical items that need to be overcome to reach our goals.
There are a large number of technologies that have been recently investigated whose main purpose has been to improve the heat transfer capabilities of devices that can be used to cool semiconductor devices that reject large quantities of heat. LHPLs continue to remain as good or better than these other devices. We will examine just a couple of the higher end sensibly cooled heat exchange technologies: microchannels and jet impingement. Microchannels require a liquid under pressure, usually pump driven and drive the liquid across a channel which extracts heat from the processors heat spreader. The contact areas that can be achieved with these devices is less than the wick areas provided by LHPLs which means to provide equivalent cooling, they need to make up for the fact that LHPLs absorb a factor of 100 as much heat per gram of coolant than they typically do. As a consequence, they end up leaving the region of the device at higher velocities and at much colder temperatures. It is also very difficult for them to provide uniform cooling across the entire heat spreader for the simple reason that they do not uniformly expose the heat spreader to a uniform flow. Jet impingement, on the other hand does expose the surface to a more uniform flow, but because of a characteristics of the way in which jets interact with surfaces along with the fact that the heated water has to be quickly removed from the region of contact, a fair amount of mixing goes on, again reducing the temperature of the resulting effluent. The heat transfer coefficient of the LHPs employed in our experiments was 0.15.degree. C./Wcm.sup.2. This state of the art performance makes it possible to cool semiconductor dies whose are is 1 inch squared and reject as much as a kilowatt. It is possible that jet impingement may be able to cool devices that reject more power, simply because of the energy that they can eject into the flow employing pumps. But, for now at least, what we have just demonstrated is that for all of the semiconductor devices that are available or likely to become available, this technology not only can reject as much energy as the competitors, but do it without requiring additional energy and at the same time producing effluents whose heat quality is excellent.
The critical role that LHPLs perform in the removal of heat from hot semiconductors, is they make it possible to remove large quantities of it, using small devices that can be packed into small locations and while at the same time providing rejection distances that make it possible to locate large efficient condensers that may employ counter-flow designs at locations in the electronic enclosure where that heat can be exchanged with either air or water. That being said, the next most important feature of the technology that this disclosure brings to the table is methods that make it possible to maintain the quality of that heat as long as possible, whether it be exchanger with air or a chilled liquid. This is a crucial part of the design approach to the heat transfer problem that we have taken.
The method we will use to greatly improve the overall coefficient of performance (COP) of the data centers cooling is by eliminating the majority of the motors typically employed to cool a data center. For this feat to be realized, it becomes important to maintain the quality of the heat being rejected by the rack cabinets that the data center may use to contain its server units.
One of the big problems in energy conservation is underestimating the important role that the quality of the heat being rejected to the final cooling device in major thermodynamic systems plays in the overall cost of buying and operating such systems. Reducing the quality of the heat too much in the case of coal fired powerplants results in the sulfuric acid condensing out in them so fast that they have to be frequently replaced. In the case of a clean large multi-megawatt fuel cell powerplant, extracting too much energy from the exhaust flow ends up driving the cost of the energy and the fan required to cool the plant up to the point where the savings get lost. The implication for data center cooling is, keep the quality of the heat up, unless you want to spend a lot of money rejecting it at the cooling tower. In existing systems, the cost of rejecting it at the cooling tower consumes 25% of the cost of running the data center, i.e., running a water chiller.
The naive approach to the use of passive heat devices suggests that like the extra cooling loops that currently consume close to 35% of the energy required to run a data center, simply stringing a series of these devices in a row, ought to be able to solve the cooling problem, without even using a cooling tower. As it turns out, a sequence of such devices will operate less efficiently than a single large Loop Heat Pipe, whose condenser line moves the heat the same distance, simply because a sequence of these devices will end up losing energy at each point of contact that connects the devices. And, since the effective driving range of the LHPs used to cool the semiconductor devices we are working with is several meters, the bottom line is that unless the cooling tower you are planning to use is in the immediate vicinity of the server you are cooling, stringing passive device together does not buy very much, but does just like the sequence of cooling loops currently employed, does dramatically degrade the heat you are attempting to reject. So, the ground rule for employing LHPLs employed to cool semiconductors turns out to be, exchange the energy with another secondary coolant, preferable one in the liquid state, as quickly as possible, if your goal is to use that coolant to drive a cooling tower directly, or to employ that heat in a cogeneration scheme or if it is simply to return air to a CRAC units heat exchanger at the highest possible temperature, thereby improving the efficiency of even an air cooled data center.
Having rejected the heat from the primary heat load in our electronic enclosure with the highest possible quality to either an air or in the case of a liquid, most likely chilled water, our goal now becomes to move it to the outside world with the smallest loss in energy. However, while doing that, we also need to consider how our LHPL primary heat removal solution interacts with the rest of the devices we use to gather up heat from the enclosure.
In data centers in which the average rack cabinet only consumed 5 KW, the fans on the rear of rack cabinets were a convenient way to help cool the contents. However, their main function at today's power levels of 20+KW, is mostly to hide the unsightly cables that drape the servers contained in the cabinet. A significant portion of the air being drawn through a typical rack cabinet ends up being drawn around the stack of server chassis within and often the asymmetric flows within the cabinet can result in eddies that circle back to the front of the cabinet near the top, heat up the top servers by as much as 15 degrees F. To get around that problem fans can be added to the top of the rack cabinet and baffles inserted between the servers and the side panels. A better way to employ such fans, is simply to insert a duct in the cabinet that can be used to gather up all the air from the rack mounted chassis and exhaust it out the rear of the cabinet by connecting it the fans on the rear door or out the top using fans mounted on the top panel or possibly to the CRAC units return air flow ducting. To make sure that this duct does what it is intended to do, a mechanism has been provided in the disclosure to seal the chassis to the duct and at the same time make sure that in the event that a chassis is not installed the duct does not leak. Furthermore, to help solve the problem of potential leaks in situations where direct chilled water is being employed within the rack cabinets, the duct can be used to contain the chilled water manifolds that serve the rack mount chassis. Finally, to make it possible for the air being removed from the rack to be reused without having to make the long trip back to the CRAC units blower, simply inserting chilled water air heat exchangers in the exit path from the rack mount chassis to the duct, makes it possible to eject the air from the cabinet at the ambient air temperature of the room. This strategy has a number of benefits of other approaches to the cooling of high power rack mount chassis that employ water cooled air heat exchangers within the rack cabinet. Besides taking up much less space in the rack cabinet, and making it possible to employ distributed heat exchangers whose total area is much larger than the ones employed by other solutions, it also reduces the total high speed fetch that the air has to make. And in the process, the amount of energy that gets injected into air flows ends up being minimized.
Reducing the energy employed moving air is one of our overall goals. When we have to do it, our goal is to move the air the smallest distance at the smallest possible speed that gets the cooling job done. The reason for this is quite simple, energy losses due to drag scale as the velocity of the air cubed multiplied by the distance it travels. Keeping the velocity and distance down, makes an enormous difference in the energy consumed by the fans driving the serves, rack cabinets and the data center itself. The technology we employ cuts down on these losses three different ways. First, when exchanging energy between air and either the primary coolant being chilled in a condenser, or a chilled liquid that is cooling it, employing finned condensers that have large areas, which our technology enables by doing things like moving the heat being rejected out of tight spaces, ends up reducing the velocity of the air required. Next, by moving air the smallest distances possible, which we make possible by picking up secondary heat sources in 1 U sealed rack mount chassis (which reduces the distance and velocity needed) or by cutting down on the distance that air needs to flow at high velocities when it is being cooled by a negative pressure duct, or by completely eliminating the need for the air to travel back to the data center's air heat exchanger, we make large reductions in the amount of energy that needs to employed by air fans and blowers. This strategy plays an important role in our energy conservation effort, and is embodied in both our sealed chassis and sealed duct designs.
The final energy reduction principle that needs to be taken into account that out embodiment improves, is water condensation. In some data centers, as much as 40% of the energy being employed by water chillers gets used to remove (by condensation) water vapor from the cooling flow which then, apparently needs to get added back into the flow to keep IT people wandering through the data center happy. It turns out that there no longer is and ESD requirement on the minimum air content of the air being employed in data centers, which basically means that keeping the relative humidity below the point where condensation occurs in the equipment can now be achieved by simply making sure the dew point of the air in the systems being cooled is less than the temperature of the liquid coolant being employed to cool systems, saving roughly 10% of the energy employed to cool some data centers, especially those in humid localities.
Our sealed chassis embodiments make this possible by keeping the dew point of the air inside of the rack mount chassis below the temperature of the coldest liquid coolant employed. This is simply accomplished in an embodiment in which we pass slightly pressurized air through a cold trap that removes excess water from it before slowly bleeding it into the “sealed” chassis, that are allowed to slowly leak air back to the ambient, at a rate that guarantees that the average air content of the chassis remains dry enough to avoid condensing if and when it comes into contact with chilled surfaces.
When it comes to cooling air cooled enclosures in general, LHPLs make it possible to make great strides in efficient uniformly distributed air cooling, by the simple act of placing the LHPL condenser at the point in the chassis where the air flow is normally exhausted out of the chassis. In the two enclosures we have studied, 1 U rack mount chassis and desktop cooled chassis, the fans that are employed on the exterior surfaces of these chassis have provided high enough flow rates to in the case of a 1 U chassis only require a single blower (already employed to pull air out of the chassis) to cool a pair of 120 Watt processors (it normally takes four to eight 1 U fans to accomplish the same task) and a single 120 mm fan running at just 1800 RPM to cool a 500 Watt CPU sitting in either the PCIe bus of the system. In all of the chassis we have examined, including the 4 U chassis employed to cool four to eight Opteron multi-core processors, the existing fans on the rear wall of the chassis that we have examined have more than enough cooling fans to make it possible to cool all of the processors, without the need for CPU fans. Which is to say, all of the chassis tested, when their CPUs were cooled using LHPLs, could get by without the need for CPU cooling fans. Not only that, the CPUs that were being cooling in situations like the 4P/8P chassis, normally require very high air flow rates even with cooling fans that fit into 2 U tall spaces simply because the CPUs in the front row end up heating the air used to cool the rear row of processors. This problem goes away with LHPLs, making it possible to actually reduce the air flow rates on the rear wall while at the same time eliminating the four to eight fans typically used to cool processors. And, while we can't claim that air cooling does as good a job as water cooling, we have gone about as far as you can go with air cooling to maintain the quality of the heat being rejected. In addition to providing sealed ducts, more uniform distribution of cooling air across the chassis and the reduction of the ambient temperature within the chassis, we have also introduced LHPL condenser designs which employ counter-flow cooling, which results in increased exit flow air temperatures which in turn end up improving the efficiency of an air cooled data center's water chiller.
When it comes to liquid cooling, the embodiments provided make it possible to employ LHPL cooling with condensers that are either directly or indirectly cooled with chilled liquids including water, safely. A new method for interfacing all closed loop passive heat transfer devices to chilled liquids has been introduced which employs a cold plate along with what we call a cold spreader (that is thermally attached to the LHPL working fluid's condenser lines) that comes into contact with the cold plate when a rack mount chassis gets installed inside of a rack cabinet. This interface, while not quite as efficient as the directly cooled interface we are about to describe, in certain situations, like blade and COTS Single Board Computer (SBC) situations, makes it possible to cool these devices as well, without using the quick disconnects that direct chilled liquids require. To improve the quality of the heat being rejected by these split condensers, a counter-flow version is also embodied and examples are provided of how to employ the cold plates that are a component of a split condenser to also cool air that is either circulating within a sealed chassis or being passed through a chassis that is being evacuated either by internal fans or a negative pressure air duct.
The most efficient cooling that we believe can be obtained using LHPLs comes when they are cooled directly with chilled liquids housed in a sealed enclosure in which the remainder of the components within the enclosure are being cooled by either liquid cooled cold plates, air that is circulating about a chassis that includes cold plates that cool it and the PCBs in the chassis and that is driven by low energy fans or blowers or air that is being circulated through the sealed chassis that passes through a chilled liquid air heat exchanger that may be a part of a component that includes the LHPL's condenser. The condenser design that we created that did the best job of producing high temperature effluent employed counter-flow heat exchange and used a chilled water jacket that was made of a material that does not readily conduct heat in addition to employing a helical wire that was thermally attached to the serpentine shaped condenser pipe, forcing the liquid to take a longer path and simultaneously increasing turbulent flow.
The final claim in the disclosure is for a data center cooled with the afore mentioned devices in which the servers in the data center room is directly attached to the cooling tower, eliminating the need for air ducting, special insulation in the walls of the data center (to keep humid air out), the need for an air blower and finally the water chiller employed by the air blower, in localities in the United States, when on the hottest most humid days of the year, an evaporative cooling tower will return water to the data center room that is at least 30 C, which is to say for most locations as hot and humid as locations like Atlanta Ga., 365 days of the year.